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ABSTRACT 

Five  similar  government  R&D  laboratories  have  been  studied 
using  "Critical    Functions  Analysis"  to  determine  the  relative 
fractions  of  their  staffs  performing  one  or  more  of  seven  critical 
functions;  a  profile  was  developed  for  each  laboratory.     In  order 
to  test  the  hypothesis  that  variations  in  these  critical   functions 
profiles  would  be  recognized  by  laboratory  users,  fifteen  different 
customers  of  these  laboratories —  weapons  system  project  managers 
and  headquarters  staff  technology  managers — were  asked  questions 
designed  to  determine  their  perceptions  of  the  relative  performance 
of  the  five  laboratories. 

Significant  variations  were  found  in  role  profiles  among  the 
several   laboratories;  important  variations  were  also  identified  in 
customers'   perceptions  about  laboratory  performance.     In  general,  the 
three  laboratories  that  had  higher  percentages  of  their  professional 
staff  performing  the  critical   functions  also  were  rated  higher  in 
performance  by  their  customers.     This  difference  was  particularly 
striking  when  an  aggregate  performance  parameter,  which  summed  the 
numerical   values  of  several   performance-sensitive  customer 
questionnaire  responses,  was  used.     Customers  were  found  to  be 
especially  sensitive  to  the  strength  of  the  laboratories' 
manufacturing/marketing  gatekeeper  functions. 
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I.   INTRODUCTION 

The  processes  that  occur  within  research  and  development 
organizations  have  been  studied  rather  extensively  over  the  past 
two  decades,  with  many  researchers  focusing  on  a  particular  function 
performed  by  key  person (s)  in  a  successfully  innovative  organization. 
For  example,  Roberts  and  Wainer  (1971)  studied  the  key  role  played 
by  the  technical  entrepreneur  in  recognizing  and  championing  a  new 
technical  idea,  product,  or  process.  Marquis  and  Rubin  (1966),  on 
the  other  hand,  chose  to  concentrate  on  the  integrative  and 
coordinative  role  played  by  the  project  manager.  Allen  (1970)  has 
studied  the  critical  part  played  by  "gatekeepers",  those  individuals 
within  the  organization  who  collect  and  distribute  information  about 
changes  in  the  internal  and  external  technical  environments.  Others, 
notably  Pelz  and  Andrews  (1966),  have  documented  the  idea  generating 
and  problem  solving  functions  of  the  creative  scientist  or  engineer. 
Finally,  Buddenhagen  (1967)  has  identified  the  role  of  the  "coach" 
or  "sponsor"  in  providing  guidance  and  development  for  less 
experienced  personnel. 

It  can  be  argued  that  these  referenced  studies  were  often 
carried  out  within  an  organizational  and  technological  environment 
such  that  focus  on  a  single  dominant  role  or  critical  function  was 
appropriate.  However,  more  usually,  particularly  when  changes  over 
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time  are  considered,  a  mix  of  activities  (and  related  critical 
functions)  must  be  performed  within  the  innovative  research  and 
development  organization.  Frohman  (1974)  and  Fusfeld  (1976)  have 
described  the  innovation  process  as  encompassing  several  comnon 
critical  functions  defined  in  Table  1. 

Insert  Table  1  Here 

Pugh-Roberts  Associates,  Inc.,  a  management  consulting  firm  in 
Cambridge,  Massachusetts,  has  developed  a  series  of  Interrelated 
questionnaires  that  are  designed  to  measure  the  number  of  professional 
staff  members  within  an  R&D  organization  who  are  performing  these 
various  critical  functions.  The  approach  used  in  the  several 
questionnaires  has  been  described  by  Frohman  and  Fusfeld  In  the 
previously  cited  references  and  it  will  not  be  repeated  In  detail 
here.  In  brief,  each  professional  employee  in  the  organization  under 
study  is  queried  to  obtain  that  individual's  assessment  of  his 
preferences,  job  requirements,  and  skills  In  performing  the  critical 
functions.  Data  are  also  obtained  from  several  co-workers  of  each 
professional  to  determine  their  perceptions  regarding  the  amount  of 
time  that  professional  devotes  to  performing  the  various  critical 
functions.  Correlation  of  the  data  from  these  two  sources  results  in 
an  inventory  of  the  current  and  potential  strengths  within  the 
organization  for  each  critical  function.  In  addition,  data  provided 
by  the  organizations'  management  regarding  goals  and  objectives  for 

the  organization  allow  the  relative  balance  or  imbalance  of  critical 
function  performance  versus  those  goals  and  objectives  to  be  assessed. 


TABLE  1 
CRITICAL  FUNCTIONS  DEFINITIONS 

1.  Problem  Solver  -  develops,  sets  up,  and  runs  experiments;  solves 
technical  problems  that  develop  as  a  project  progresses 

2.  Idea  Generator  -  contributes  to  knowledge  in  technical  field; 
finds  opportunities  for  technical  solutions  to  marketing, 
technical,  or  customer  service  problems 

3.  Technical  Information  Gatekeeper  -  keeps  abreast  of  technical 
developments  both  inside  and  outside  of  the  organization;  serves 
as  a  source  of  technical  information  for  colleagues 

4.  Marketing/Manufacturing  Information  Gatekeeper  -  maintains 
familiarity  with  existing  and  potential  markets,  whether  outside 
customers  or  inside  manufacturing  divisions;  communicates  market 
information  to  R&D  so  that  products  or  processes  under  development 
appropriately  consider  marketplace  and  manufacturing  requirements 

5.  Entrepreneur  -  the  real  motivator  behind  a  project;  possesses  the  drive 
and  determination  to  sell  a  new  product  to  management  and  obtain 
resources  to  get  the  project  started 

6.  Project  Manager  -  coordinates,  plans,  and  schedules  work  for  a 
project;  serves  as  the  link  between  management  and  R&D  who  balances 

the  company's  goals  with  those  of  R&D 

7.  Coach  -  provides  less  experienced  staff  with  credibility,  support 
and  guidance 


II.  RESEARCH  APPROACH 

The  hypothesis  to  be  tested  in  this  research  was  that  significant 
differences  in  performance  among  research  and  development  organizations 
could  be  related  to  significant  differences  in  their  skill  inventories 
(as  determined  by  critical  functions  analysis).  To  test  this 
hypothesis,  one  had  to  find  research  and  development  organizations  with 
sufficiently  similar  responsibilities  and  functions,  inventory  skills 
of  their  staffs,  and  also  obtain  independent  performance  measurements 
for  the  same  organizations.  Differences  in  these  attributes  could  then 
be  examined  for  correlation  and  significance,  and  conclusions  drawn. 

It  is  suggested  that  such  an  experiment  could  be  difficult  to 
structure  in  the  private  sector  due  to  the  problem  of  finding  several 
R&D  organizations  with  similar  external  relationships,  while,  at  the 
same  time,  finding  a  way  to  objectively  measure  their  performance. 
Fortunately,  it  was  fairly  easy  to  find  this  situation  in  the  federal 
government. 

DESCRIPTION  OF  ORGANIZATIONS  STUDIED 

The  organizations  chosen  for  this  study  were  five  U.S.   Department 
of  Defense  in-house  laboratories.     These  five  laboratories  vary 
considerably  in  the  size  of  their  professional  work  force,  as  shown 
in  Table  2,  but  are  similar  in  internal  organization  design,  and  in 
their  relationships  with  the  outside  world. 

Insert  Table  2  Here 


TABLE  2 
LABORATORY  CHARACTERISTICS 


LAB 

% 
PROFESSIONALS 

#  OF 
PROFESSIONALS 

#  IN 
THIS  STUDY 

% 
PARTICIPATION 

A 

63 

149 

129 

87 

B 

74 

81 

65 

82 

C 

74 

69 

63 

91 

D 

71 

110 

99 

90 

E 

57 

47 

47 

100 

TOTALS         —  456  403         88 


Each  of  the  laboratories   is  responsible  for  work  (predominantly 
applied  research  in  nature)  in  a  particular  technology  area;  each  of 
these  areas  encompasses  the  technology  underlying  a  major  component 
of  a  weapons  system.     Approximately  one-half  of  each  laboratory's 
multi-million  dollar  annual   cash  flow  is  devoted  to  in-house  and 
contracted  effort  to  advance  technology  in  its  area  of  responsibility 
to  meet  future  U.S.   defense  needs.     The   "customers"  for  these  R&D 
technology  efforts  are  senior  technical  managers  within  the  Defense 
Department  (and,  ultimately,  Congress). 

The  other  half  of  each  laboratory's  funds  come  from  a  second 
set  of  "customers",  the  managers  of  weapons  systems  under 
development  or  in  production.     For  these  customers,  the  laboratories 
provide  research  and  development  services.     The  laboratories  studied 
had  customer  relationships  with  as  many  as  eleven  different  weapons 
system  managers. 

Each  laboratory  is  managed  by  a  Director  who  is  supported  by 
a  small   administrative  staff.     Reporting  to  the  Director  are  several 
supervisors,  each  of  whom  is  responsible  for  the  direction  of  a  group 
of  professional    (and  non-professional)  employees  in  a  functional 
area.     When  warranted,  employees  can  be  moved  from  one  functional   area 
to  another.     At  any  time,  many  tasks  are  underway  in  a  laboratory, 
frequently  involving  more  than  one  functional  area.     As  a  consequence, 
the  laboratories  operate  primarily  in  a  matrix  management  mode; 
designated  project  leaders  have  responsibility  for  the  direction  of 
individual   research  and  development  tasks  within  the  laboratory.     These 
project  leaders  might  also  be  functional   supervisors,  but  frequently 


are  not;  they  are  usually  senior  professional   personnel,  however.     In 
addition  to  the  similarities  described  above,  the  relative  fractions 
of  junior  and  senior  professional  employees  are  approximately  the  same 
for  each  laboratory.     As  shown  in  Table  2,  the  ratio  of  professionals  to 
total   laboratory  personnel   does  vary  from  laboratory  to  laboratory, 

reflecting  primarily  the  specific  demands  of  the  technology  area  for 
technicians  and  other  sub-professional  personnel. 

CUSTOMERS  AND  LABORATORY  PERFORMANCE 

The  customers  of  the  laboratories  were  chosen  as  the  appropriate 
judges  to  provide  perceptions  on  laboratory  skills  and  performance. 
All   the  laboratories  had  four  R&D  technology  customers  in  common,  as 
well   as  at  least  seven  of  the  eleven  R&D  services  customers.     The 
questionnaires  developed  by  Pugh-Roberts  Associates,  with  minor 
modifications  to  make  the  terminology  consistent  with  the  government 
laboratory  environment,  were  used  to  determine  the  laboratory  role 
profiles.     A  new  questionnaire,  entitled  "Customer  Perceptions 
Instrument",  was  designed  and  used  to  gather  customer  data.     It 
consisted  of  eleven  questions  and  was  designed  based  on  the  following 
concepts . 

First,  the  questionnaire  was  designed  to  focus  in  main  on  those 
laboratory  attributes  which  could  be  related  to  the  relative  performance 
or  lack  of  performance  of  one  or  more  of  the  critical   functions.     For 
example,  customers  were  asked,   "How  well   does  this  laboratory  under- 
stand the  needs  of  your  organization?"     Laboratories  receiving  a  high 
score  on  this  question   (i.e.   that  understood  the  customers'  needs  y/ery 


well)  would  be  expected  to  exhibit  relative  strength  in  the 
manufacturing  market/gatekeeper  function.  A  total  of  five  questions 
were  designed  based  on  this  philosophy. 

A  second  major  area  for  focus  concerned  overall  laboratory 
performance  measures. 

Four  questions  were  used  to  explore  laboratory  innovativeness ,  to 
address  the  issue  of  usefulness  and  importance  of  the  laboratory's 
output  (or  "products"),  and  to  obtain  a  comparison  of  the  laboratory 
with  other  research  and  development  organizations. 

Finally,  two  questions  were  designed  to  provide  data  for 
comparison  with  that  obtained  from  the  laboratory  management  teams  on 
the  status  of  their  technology  areas  and  the  emphasis  and  focus  of 
their  professional  staffs. 

ANALYSIS  OF  RESULTS 

Customer  Perceptions 

The  responses  received  to  the  eleven  questions  of  the  Customer 
Perceptions  Instrument  indicate  that  the  several  customers  of  the 
laboratories  did  recognize  significant  differences  between  them.  For 
example,  the  responses  to  the  "understand  your  needs"  question 
(Question  2)  are  given  in  detail  in  Table  3,  with  the  customer  sub- 
groupings  varied.  This  variation  was  carried  out  for  each  question  in 


Insert  Table  3  Here 


order  to  investigate  whether  the  two  classes  of  customers  ("R&D 
services"  and  "R&D  technology")  had  significantly  different 
perceptions.  A  second  variation  was  accomplished  to  investigate  the 


TABLE  3 
CUSTOMER  RESPONSES  TO  QUESTION  NUMBER  Tl^O 
"How  well   does   this   laboratory  understand  the  needs  of  your  organization?" 

RESPONSES   FOR  LABORATORY 


A 

B 

C 

D 

E 

All   Customers   (15) 

3.90 

4.73 

3.82 

3.62 

2.75 

In-Conmon  Customers 

(8) 

4.00 

5.00 

4.25 

3.62 

2.75 

In-Common  R&D 

Services  Customers 

(7) 

3.86 

4.86 

3.86 

4.43 

2.29 

NOTES:     1.     The  responses  for  E  are  significantly  different  from  those 
for  B,  with  99+  percent  confidence. 

2.  The  responses  for  E  are  significantly  different  from  those 
for  A  and  C,  with  95+  percent  confidence. 

3.  The  responses  for  D  are  significantly  different  from 
those  of  B,  with  95+  percent  confidence. 

4.  Differences  based  on  Aspin-Welch  two-sided  t-test. 

5.  Scores  could  range  from  1    (good)  to  7  (bad).     Mean  values 
are  reported  here. 


possible  effects  of  the  asymmetry  in  customers  among  laboratories. 
Recalling  from  the  rationale  previously  presented  that  responses  to 
Question  2  were  expected  to  reflect  differences  in  laboratory 
marketing  gatekeeper  role  performance,  the  data  in  Table  3  would  lead 
us  to  anticipate  with  high  confidence  that  laboratory  E  would  have  a 
higher  percentage  of  market  gatekeepers  than  laboratory  B.  With 
somewhat  lower  confidence  we  would  predict  E  to  have  a  higher 
percentage  of  market  gatekeepers  than  A  or  C,  and  about  the  same 
number  as  D.  Similarly,  D  is  expected  to  have  more  market  gatekeepers 
than  does  B. 

As  a  second  example,  Question  4  was  designed  with  the  idea  that 
differences  would  reflect  the  degree  to  which  entrepreneurial  role 
players  were  present.  Table  4  presents  the  results  for  this  question 
in  the  same  format  as  was  used  in  Table  3.  Here  the  differences  are  not 

Insert  Table  4  Here 

so  consistently  significant;  about  all  we  can  expect  is  that  laboratory 
E  will  have  a  somewhat  higher  percentage  of  entrepreneurs  than  do 
laboratories  B  or  C.  It  may  be  that  customers  in  responding  to  this 
question  are  sensitive  to  more  than  just  entrepreneurial  (the  willingness 
to  undertake  unfunded  effort)  performance;  for  example,  the  quality  of 
the  effort  performed  may  influence  the  response. 

Four  of  the  questions  asked  each  customer  can  be  regarded  as 
different  ways  of  asking  the  same  question  about  the  laboratories' 
overall  performance  in  innovation;  an  aggregated  approach  to  analyzing 
these  responses  was  selected.  A  simple  "effectiveness  parameter"  was 


TABLE  4 
RESPONSES  TO  QUESTION  NUMBER  4 
"Evaluate  the  responsiveness  of  this   laboratory  in  conducting  programs 
that  are  not  funded  by  you  to  meet  your  needs." 

RESPONSES  FOR  LABORATORY 
ABC  D  E 

All   Customers   (11)  4.67         5.50         5.00  4.00  3.50 

In-Common  Customers   (7)  5.14         5.71         5.57  4.00  3.50 

NOTES:     1.      In  this  case,  the  "in-common  customers"  and  the  "in-cormion 
R&D  services  customers"  are  identical. 

2.  The  responses  for  E  are  significantly  different  from  those 
of  B  or  C  with  90+  or  95+  percent  confidence. 

3.  Differences  based  on  Aspin-Welch  two-sided  t-test. 

4.  Scores  could  range  from  1   (good)  to  7  (bad).     Mean  values 
are  reported  here. 


devised  to  provide  this  aggregation,  and  was  obtained  by  subtracting 
the  (numerical)  mean  responses  to  the  four  questions  from  a  constant. 

The  constant  used  was  the  maximum  value  the  aggregated  responses  could 

take  on  and  was  introduced  to  compensate  for  the  choice  of  response 

scales,  which  would  lead  to  "low  effectiveness  is  good"  results  without 

this  adjustment.  Values  of  this  effectiveness  parameter  are  plotted  in 
Figure  1  for  several  combinations  of  customers.  The  results  are 

Insert  Finure  1  Here 

consistent  and  show  that,  independent  of  the  sub-grouping  of 
customers  that  is  selected,  the  parameter  values  for  laboratories  A, 
D  and  E  are  not  significantly  different.  The  same  can  be  said  for  the 
values  for  laboratories  B  and  C.  However,  the  values  for  these  two 
groupings  of  laboratories  are  significantly  (p  ^  .01)  different,  with 
the  A-D-E  grouping  rated  as  considerably  higher  in  effectiveness. 

One  additional  result  from  the  customer  perceptions  data  is  worth 
noting.  Question  1  was  identical  to  a  question  asked  the  laboratory 
management  teams;  the  results  from  these  two  groups  of  respondents  are 
given  in  Table  5.  We  observe  no  significant  difference  between  the 

Insert  Table  5  Here 

perceptions  of  customers  and  laboratory  management  for  laboratories  A, 
D  and  E  concerning  the  status  of  their  technology.  (While  not 
statistically  significant,  it  is  interesting  to  note  that  the 
managements  of  these  three  laboratories  in  general  regarded  their 
technology  areas  as  less  dynamic  or  less  advancing  than  did  their 
customers).  However,  in  the  case  of  laboratories  B  and  C,  the 
difference  is  strongly  significant.  These  two  laboratory  management 
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FIGURE  1 
LABORATORY  EFFECTIVENESS  PARAMETER  COMPARISON 
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TABLE  5 
RESPONSES  TO  QUESTION  NUMBER  ONE 
"Describe  the  rate  of  change  in  the  technological   area  in  which  this 
laboratory  works."  RESPONSES   FOR  LABORATORY 

A  B  C  D  E 

Laboratory  Management  4.79       4.50         5.50  5.16         3.71 

(Number  of  Respondents)  (9)         (2)  (4)  (6)  (7) 

All    Customers   (15)  5.38       2.43*       3.21*         5.55         3.87 

In -Coimon  Customers   (11)  5.18      2.45*       3.09*         5.55         3.91 

R&D  Technology  Customers   (4)  5.00       2.25*      2.75*        6.25         3.50 

In-Common  R&D  Service 

Customers   (7)  5.56       2.50*       3.40*         5.14         4.00 

*Significant  at  99+  percent  level 
(Aspin-Welch  two-sided  t-test) 
NOTE:     The  higher  the  score  from  1    to  7,  the  more  rapidly  advancing 
the  technology.     Mean  values  are  reported  here. 


teams  regarded  their  technology  areas  as  considerably  more  dynamic  and 
advancing  than  did  their  customers.  A  caveat  regarding  these  results 
is  appropriate,  however;  the  number  of  laboratory  management  respondents 
for  these  two  laboratories  is  small,  even  though  the  variance  of  these 
few  responses  is  also  small. 

LABORATORY  ROLE  PROFILES 

Inter-laboratory  comparisons  are  shown  in  Figure  2.  Note  that 

Insert  Figure  2  Here 

in  general  the  "curves"  for  laboratories  A,  D  and  E  lie  above  those 

for  B  and  C.   In  particular,  laboratory  E  shows  a  stronger  marketing/ 

manufacturing  gatekeeper  function  (twice  the  percentage) 
than  laboratories  B  or  C.  These  profiles  were  developed  using 

a  screening  criterion  of  one  standard  deviation  above  the  mean, 

following  Swain  (1971).  That  is,  a  professional  was  counted  as 

performing  a  particular  critical  function  if  analysis  of  the  several 

questionnaires  showed  that  he  performed  it  to  a  greater  degree  than 

(approximately)  85  percent  of  his  peers.  Profiles  were  also 

developed  using  two  standard  deviations  above  the  mean  and  one-half 

standard  deviation  above  the  mean  as  screening  criteria.  The  same 

trends  (between  laboratories)  were  observed  as  with  the  one  standard 

deviation  criterion  shown  here. 

IV.  CONCLUSIONS 

The  results  of  the  preceding  data  analysis  provide  the  basis  for 

several  conclusions.  First,  a  significant  correlation  was  found 

between  differences  in  laboratory  role  profiles  (as  shown  in  Figure  2) 

and  differences  in  customer  perceptions  about  laboratory  performance 


FIGURE  2 
ROLE  PROFILES  -  INTER- LABORATORY  COMPARISON 
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(Figure  1).  Those  laboratories  that  show  consistently  higher  percentages 
of  professionals  performing  each  of  the  critical  functions  have 
significantly  higher  values  of  effectiveness,  at  least  as  measured 
by  the  simple  effectiveness  parameter  devised  for  this  study. 

A  significant  correlation  also  was  found  between  customer 
perceptions  and  the  strength  of  the  manufacturing/marketing  gatekeeper 
role  --  the  higher  the  percentage  of  individuals  performing  the  role, 

the  better  the  customers  perceived  that  the  laboratory  understood 
their  needs. 

Another  facet  of  performance  of  the  manufacturing/marketing 
function  surfaced  when  the  responses  of  customers  and  laboratory 
management  to  the  (identical)  question  of  status  of  the  technology 
area  were  compared.  Very  significant  differences  were  found  between 
laboratory  and  customer  perceptions  about  that  status  for  those  two 
laboratories  which  have  low  percentages  of  manufacturing/marketing 
performance;  specifically,  in  these  cases  the  customers  felt  that 
the  technology  area  was  much  more  slowly  advancing  than  did  laboratory 
management. 

Two  of  the  laboratories  (A  and  D)  work  in  technology  areas  in 
which  advances  are  occurring  rapidly  (as  perceived  by  both  their 
customers  and  laboratory  management).  One  might  expect  to  find  a 
significantly  greater  fraction  of  entrepreneurs  in  such  a  situation; 
such  was  in  fact  found.  These  laboratories  had  one  and  one-half  to 
two  times  the  percentage  of  professionals  performing  the  entrepreneurial 
function  compared  with  the  laboratories  working  in  more  slowly 
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advancing  areas. 

The  two  subsets  of  laboratory  customers—  "R&D  Technology" 
and  R&D  Services"—  did  not  significantly  vary  in  their  perceptions 
concerning  laboratory  performance.     In  some  cases,  the  statistical 

significance  varied,  but  in  no  case  did  the  trends  change;  in  short, 
both  classes  of  customers  saw  the  laboratories  in  the  same  light. 

In  summary,  there  are  significant  correlations  between 
differences  in  laboratory  role  profiles  and  customer  perceptions 
concerning  laboratory  performance,  as  hypothesized.     These  results 
strongly  suggest  that  if  one  uses  critical   functions  analysis  to 
determine  staffing  balance  and  needs,  one  can  do  so  with  assurance 
that  the  organization's  customers  are  in  fact  sensitive  to  balances 
or  imbalances  in  critical   functions  performers.     The  underlying 
assumption  here  is  that  customers  are  vital   to  the  economic  and 
professional  well-being  of  innovative  research  and  development 
organizations. 
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